The Effect of Collection Organizationandquery Locality on Informationretrieval

The explosion of content in distributed information retrieval (IR) systems requires new mechanisms in order to attain timely and accurate retrieval of unstructured text. Collection selection and partial collection replication with replica selection are two such mechanisms that enable IR systems to search a small percentage of data and thus improve performance and scalability. To maintain eeectiveness simultaneously, IR systems must be conngured carefully, and consider workload locality, possible collection organizationscollection organization, and any interaction that results. This work builds on previous results which have focused on maintaining eeectiveness. We propose IR system architec-tures with collection selection and partial replication based on collection organization and query locality characteristics which maintain accuracy and achieve high performance. We compare conngurations using a validated simulator that partition data and replicate data, and their sensitivities to workload locality, collection organization, collection size, and hardware. Locality and collection organization have dramatic eeects on performance. For example, we demonstrate with simulation results that collection selection performs especially well when the distribution of queries to collections is uniform and collections are organized by topics , but it suuers when particular collections are \hot." We nd that when queries have even modest locality, conngurations that replicate 1 2 data outperform those that partition data, usually signiicantly. These results point to good IR system designs under a variety of workloads and collection organizations.